Method and system for synchronization of content rendering

ABSTRACT

A method and system for synchronizing the rendering of content at various rendering devices. Each rendering device has a device time and a rendering time. The synchronization system designates one of the rendering devices as a master rendering device and designates all other rendering devices as slave rendering devices. Each slave rendering device adjusts the rendering of its content to keep it in synchronization with the rendering of the content at the master rendering device. The master rendering device sends a message with its rendering time and corresponding device time to the slave rendering devices. Each slave rendering device, upon receiving the message from the master rendering device, determines whether it is synchronized with the master rendering time. If not, the slave rendering device adjusts the rendering of its content to compensate for the difference between the master rendering time and the slave rendering time.

PRIORITY CLAIM

This application is a continuation of U.S. application Ser. No.11/933,194, filed Oct. 31, 2007 now abandoned, which is a continuationof U.S. application Ser. No. 10/322,335, filed Dec. 17, 2002, now U.S.Pat. No. 7,391,791, which claims the benefit of U.S. ProvisionalApplication No. 60/341,574, filed Dec. 17, 2001.

TECHNICAL FIELD

The described technology relates to rendering of content at multiplerendering devices in a synchronized manner.

BACKGROUND

The following application is incorporated by reference as if fully setforth herein: U.S. application Ser. No. 10/322,335 filed Dec. 17, 2002.

Multimedia presentations that are presented on different renderingdevices (e.g., video display and stereo system) typically require thatthe different content of the presentation be rendered in a synchronizedmanner. For example, a multimedia presentation may include video, audio,and text content that should be rendered in a synchronized manner. Theaudio and text content may correspond to the dialogue of the video.Thus, the audio and text contents need to be rendered in a synchronizedmanner with the video content. Typically, the content of a multimediapresentation is stored at a single location, such as on a disk drive ofa source device. To render the presentation, the source device retrieveseach different type of content and sends it to the appropriate renderingdevice to effect the multimedia presentation. The source device thensends the content to the rendering devices in sufficient time so thatthe rendering devices can receive and render the content in a timelymanner.

Various rendering devices, however, may have different time domains thatmake the rendering of the multimedia presentation in a synchronizedmanner difficult. For example, video and audio rendering devices mayhave system clocks that operate at slightly different frequencies. As aresult, the video and audio content will gradually appear to the personviewing the presentation to be out of synchronization. The rendering ofcontent in a synchronized manner is made even more difficult becausesome rendering devices may have multiple time domains. For example, anaudio rendering device may have a system clock and a clock on a digitalsignal processing (“DSP”) interface card. In such a case, thecombination of clocks may result in the presentation becoming even morequickly out of synchronization.

It would be desirable to have the technique that would facilitate therendering of the multimedia presentation in a synchronized manner.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating synchronization of renderingdevices in one embodiment.

FIG. 2 is a diagram illustrating the calculation of the time domaindifferential between two devices.

FIG. 3 illustrates a time domain table for a rendering device in oneembodiment.

FIG. 4 illustrates a block diagram of another embodiment of thesynchronization system.

FIG. 5 is a block diagram illustrating components of a content renderingdevice in one embodiment.

FIG. 6 is a flow diagram illustrating the processing of the send timedomain message component in one embodiment.

FIG. 7 is a flow diagram of the receive time domain message component inone embodiment.

FIG. 8 is a flow diagram illustrating the render content component inone embodiment.

FIG. 9 is a flow diagram illustrating the process of the send renderingtime message component in one embodiment.

FIG. 10 is a block diagram illustrating the processing of the receiverendering time message component in one embodiment.

DETAILED DESCRIPTION

A method and system for synchronizing the rendering of content atvarious rendering devices is provided. In one embodiment, each renderingdevice has a device time and a rendering time. The device time is thetime as indicated by a designated clock (e.g., system clock) of therendering device. The rendering time is the time represented by theamount of content that has been rendered by that rendering device. Forexample, if a rendering device is displaying 30 frames of video persecond, then the rendering time will be 15 seconds after 450 frames aredisplayed. The rendering time of content at a rendering device has a“corresponding” device time, which is the device time at which therendering time occurred. For example, the rendering time of 15 secondsmay have a corresponding device time of 30 minutes and 15 seconds whenthe rendering device initialized 30 minutes before the start ofrendering the video. To help ensure synchronization of renderingdevices, the synchronization system designates one of the renderingdevices as a master rendering device and designates all other renderingdevices as slave rendering devices. Each slave rendering device adjuststhe rendering of its content to keep it in synchronization with therendering of the content at the master rendering device. The masterrendering device sends a message with its rendering time andcorresponding device time to the slave rendering devices. Each slaverendering device, upon receiving the message from the master renderingdevice, determines whether it is synchronized with the master renderingtime. If not, the slave rendering device adjusts the rendering of itscontent to compensate for the difference between the master renderingtime and the slave rendering time. A slave rendering device candetermine the amount it is out of synchronization by comparing its slaverendering time at a certain slave device time to the master renderingtime at that same device time. Alternatively, the amount can bedetermined by comparing its slave device time at a certain renderingtime to the master device time at that same rendering time. In anotherembodiment, the synchronization system can define a default renderingtime for the synchronization. In such a case, the master renderingdevice need only include its effective device time that corresponds tothe default rendering time in the message that is sent to the slaverendering devices. For example, the default rendering time might be therendering time of zero. In such a case, the master rendering device cansubtract its current rendering time from its current device time to giveits effective device time at rendering time zero. A slave renderingdevice, knowing the default rendering time, can determine whether it issynchronized and the variation in rendering time between the masterrendering device and the slave rendering device.

In one embodiment, the synchronization system allows for two sources ofcontent to be synchronized even though the rendering times of thesources are not themselves synchronized. For example, two separatesources may be video transmitted via satellite and audio transmitted vialand telephone lines. If audio is being transmitted and then a fewseconds later the corresponding video starts to be transmitted, then therendering times of zero for the audio and video will not correspond to asynchronized state. For example, the video at the video rendering timeof zero should be rendered at the same time as the audio with the audiorendering time of five is rendered. This difference in rendering timesis referred to as source offset. In addition, the difference in thepropagation delay resulting from the different transmission paths of thevideo and audio may be variable and thus contribute to a variation insynchronization that is variable and is not known in advance.

To account for this lack of synchronization, the synchronization systemallows a user (e.g., the person viewing the content) to manually accountfor the variation. For example, if the video and audio are rendered viaa personal computer, the synchronization system may display a dial or aslider on a user interface that the user can adjust to indicate thedifference in the rendering times. If the video is rendered five secondsafter the corresponding audio, then the user can indicate via the userinterface that the offset is five seconds. In such a case, thesynchronization system may use the offset to adjust the rendering timeof the audio so that the audio associated with the adjusted audiorendering time should be rendered at the same time as the video contentwith the same video rendering time. The synchronization system couldbuffer the audio to account for the offset.

The synchronization system in one embodiment factors in the differencesin the time domains of the various rendering devices when evaluatingsynchronization. The rendering devices exchange device time informationso that the rendering devices can account for the differences in thetime domains of the other rendering devices. Each rendering device maysend to the other rendering devices a time domain message that includesits current device time (i.e., send time) along with the time itreceived the last time domain message (i.e., receive time) from each ofthe other rendering devices and the send time of that last time domainmessage. When a rendering device receives such a time domain message, itcalculates the time differential between its time domain and the timedomain of the sending rendering device. In one embodiment, thesynchronization system calculates the time domain differential bycombining the difference in send and receive times for the last messagessent to and received from another device in a way that helps factor outthe transmission time of the messages. A slave rendering device can thenuse this time domain differential to convert the master device time tothe time domain of the slave when synchronizing the rendering ofcontent. In one embodiment, each rendering device broadcasts at varioustimes its time domain message. The time domain message includes areceived time for a message received for each of the other renderingdevices. Each rendering device receives the broadcast time domainmessage. The receiving rendering device can then calculate its timedomain differential with the broadcasting rendering device. In this way,time domain differentials can be determined on a peer-to-peer basiswithout the need for a master device to keep a master time and bybroadcasting the time domain messages, rather then sending separate timedomain messages for each pair of devices.

FIG. 1 is a block diagram illustrating synchronization of renderingdevices in one embodiment. The source device 101 distributes the contentof a presentation to the video rendering device 102, the audio renderingdevice 103, and the text rendering device 104 via communications link105. The source device may have the multimedia presentation storedlocally, for example, on a disk drive, may dynamically generate themultimedia presentation, may receive content of the multimediapresentation from other sources, and so on. A multimedia presentation isany presentation that includes different content that is to be renderedin a synchronized manner. For example, the content could be video andaudio content for a virtual ride in a theme park along with motioncontent to control the ride. As another example, the presentation mayinclude light content that controls the display of a laser to besynchronized with audio content. Also, the “synchronization” of contentmay be different for different rendering devices. For example, audiocontent may be sent to multiple audio rendering devices with theexpectation that some of the audio rendering devices may delay renderingfor a certain period (e.g., 10 milliseconds) to achieve a desired audioeffect. In such a case, the rendering is considered synchronized whenthe delay equals that period. The synchronization system designates oneof the rendering devices as the master rendering device. In thisexample, the audio rendering device 103 is designated as the masterrendering device, and the video rendering device 102 and text renderingdevice 104 are designated as slave rendering devices. After the sourcestarts sending the content to the rendering devices, the audio renderingdevice broadcasts a master rendering time message with its master devicetime and master rendering time to the slave rendering devices on aperiodic basis. In this example, since the communications link ispoint-to-point from the source device, the audio rendering device sendsthe message to the source device, which in turn forwards the message tothe slave rendering devices. Upon receiving the master rendering timemessage, the slave rendering devices convert the master device time totheir own time domains and then calculate the difference between theirslave rendering time and the master rendering time at a certain point intime. In one embodiment, the synchronization system uses a device timeat a calculated start of sending as the point in time. The slaverendering devices then adjusts the rendering as appropriate tocompensate for the difference. The rendering device adjusts therendering of their content in ways that are appropriate for theircontent. For example, if the video rendering device was one secondbehind the master audio rendering device, then it might skip the displayof every other frame for the next two seconds to “speed up” to themaster audio rendering device. Alternately, if the video renderingdevice was one second ahead of the master audio rendering device, thenthe video rendering device might display each of the next 30 framestwice to “slow down” to the master audio rendering device.

FIG. 2 is a diagram illustrating the calculation of the time domaindifferential between two devices. Device 1 initially sends to device 2 atime domain message 301 that includes its current device time, referredto as “sendtime1.” When device 2 receives the time domain message, itstores the sendtime1 along with the time it received the time domainmessage, referred to as “receivetime1.” Device 2 then sends to device 1a time domain message 302 that includes its device time, referred to as“sendtime2,” along with sendtime1 and receivetime1. When device 1receives the time domain message, it stores sendtime1, receivetime1, andsendtime2 along with its device time, referred to as “receivetime2.”Device 1 now has enough information to calculate the time domaindifferential according to the following formula:Diff=((RT1−ST1)+(ST2−RT2))/2where Diff is the time domain differential, RT is receive time, and STis send time. Device 1 then sends a time domain message 303 to device 2that includes its device time, referred to as “sendtime3” along withsendtime2 and receivetime2. When device 2 receives the time domainmessage, it stores sendtime2, receivetime2, and sendtime3 along with itsdevice time, referred to as “receivetime3.” Device 2 now has enoughinformation to calculate the time differential according to a similarformula.

This formula calculates the difference between the send time and thereceive time for time domain messages between the two devices. If therewas no variation in the time domains between the devices, then the sendand receive times would reflect the communications link latency betweensending and receiving the time domain messages. In one embodiment, thesynchronization system assumes that the latency in transmitting amessage from one device to another device is approximately the same asthe latency in transmitting the message from the other device to thedevice. Thus, the synchronization system calculates the time domaindifference by taking the average of the differences in the send andreceive times of the messages. The receive time of the messages isrepresented by the following equations:RT1=ST1+Diff+LRT2=ST2−Diff+Lwhere Diff represents the time domain differential and L represents thelatency of the communications link. These equations are equivalent tothe following equations:Diff=RT1−ST1−LDiff=ST2−RT2+LThe average of these two equations isDiff=((RT1−ST1−L)+(ST2−RT2+L))/2The latency factors out of the equation to give the following equation:Diff=((RT1−ST1)+(ST2−RT2))/2

FIG. 3 illustrates a time domain table for a rendering device in oneembodiment. The time domain table of a device includes a row for eachother device to which the device is connected. For example, the audiorendering device 103 of FIG. 1 would have a row for the source device101, the video rendering device 102, and the text rendering device 104.In this example, the time domain table includes a node identifier column301, a sendtime1 column 302, a receivetime1 column 303, a sendtime2column 304, a receivetime2 column 305, and a time domain differentialcolumn 306. A positive time domain differential indicates the number oftime units that this device is ahead of the other device, and a negativetime domain differential indicates the number of time units that thisdevice is behind the other device. Thus, in this example, the devicetime of the audio rendering device 103 is ahead of the device time ofthe source device 101 by 1000 time units. In contrast, the device timeof the audio rendering device 103 is behind the device time of the videorendering device 102 by 495 time units. One skilled in the art willappreciate the time units can be any units appropriate to the desiredsynchronization accuracy, such as milliseconds, microseconds, and so on.One skilled in the art will appreciate the time domain messages need notinclude the times set by the receiving device. For example, the timedomain message 302 need not include sendtime1 since device 1 could havestored that time locally.

FIG. 4 illustrates a block diagram of another embodiment of thesynchronization system. In this example, the source device 400 performsthe function of the master, and the video rendering device 401, theaudio rendering device 402, and the text rendering device 403 areslaves. In particular, the source device, even though it does norendering itself, may keep track of an idealized rendering time that maynot correspond to the actual rendering time of any of the renderingdevices. In such a situation, the master source device periodicallysends a rendering time message that includes its device time along withthe corresponding idealized rendering time to each of the renderingdevices. The rendering devices can then adjust their rendering in thesame manner as if the rendering time message is sent from a masterrendering device. Alternatively, each rendering device can provide theirdevice time and corresponding rendering time to the source device. Thesource device can then calculate the rendering time differential foreach rendering device and provide that differential to the renderingdevices to speed up or slow down their rendering as appropriate.

FIG. 5 is a block diagram illustrating components of a content renderingdevice in one embodiment. The content rendering device 500 includes areceive content component 501, a render content component 502, a sendtime domain message component 503, a receive time domain messagecomponent 504, a time domain table 505, a send rendering time messagecomponent 506, and a receive rendering time message component 507. Thereceive content component receives content from the source device andmay store the content in a buffer for subsequent rendering. Therendering content component retrieves the buffered content and effectsthe rendering of the content. The send time domain message componentsends time domain messages to the other devices. The send time domainmessage component may send the message upon occurrence of an event, suchas when a timer expires, when a message is received, and so on. Oneskilled in the art will appreciate that the frequency of sending timedomain messages can be adjusted to account for the anticipated driftbetween clocks of the rendering devices. The receive time domain messagecomponent receives the time domain messages sent by other devices andupdates the time domain table as appropriate. The send rendering timemessage component is used when this content rendering device is a masterrendering device to send a rendering time message to the other renderingdevices. The receive rendering time message component receives therendering time messages sent by the master device and calculates arendering time differential that is used to adjust the rendering of thecontent. The devices may include a central processing unit, memory,input devices (e.g., keyboard and pointing devices), output devices(e.g., display devices), and storage devices (e.g., disk drives). Thememory and storage devices are computer-readable media that may containinstructions that implement the synchronization system. In addition,data structures and message structures may be stored or transmitted viaa data transmission medium, such as a signal on a communications link.Various communications links may be used, such as the Internet, a localarea network, a wide area network, or a point-to-point dial-upconnection.

FIG. 6 is a flow diagram illustrating the processing of the send timedomain message component in one embodiment. In block 601, the componentadds the identifier of this device to the time domain message. In block602, the component adds the send time to the message. The send time isthe current device time. In blocks 603-607, the component loopsselecting each other device and adding times for that device to the timedomain message then loops to block 603 to select the next device. Inblock 608, the component sends the time domain message to the otherdevices and then completes.

FIG. 7 is a flow diagram of the receive time domain message component inone embodiment. In decision block 701, if the identifier of this deviceis in the list of device identifiers in the message, then the componentcontinues at block 702, else the component completes. In block 702, thecomponent retrieves the current send time from the message and saves itin the time domain table. In block 703, the component retrieves the lastsend time from the message and saves it in the time domain table. Inblock 704, the component retrieves the last receive time from themessage and saves it in the time domain table. In block 705, thecomponent retrieves the device time as the current receive time andsaves it in the time domain table. The time values may be saved bystoring them in the time domain table in the row associated with thedevice that sent the message. In block 706, the component calculates thetime domain differential. In block 707, the component smoothes the timedomain differential. The time domain differential can be smoothed usingvarious techniques such as averaging the last several time domaindifferentials using a decaying function to limit the impact of theoldest time domain differentials. In one embodiment, the synchronizationsystem saves the values of the last eight pairs of time domaindifferentials (i.e., ST2−RT2 and RT1−ST1) and uses the average of theminimum value of the set of eight larger differentials and the maximumvalue of the set of eight smaller differentials as the time domaindifferential. The component then completes.

FIG. 8 is a flow diagram illustrating the render content component inone embodiment. In blocks 801-806, the component loops processing eachblock of content that is received from the source device. In block 801,the component selects the next block of content provided by the sourcedevice. The content may be buffered at this rendering device. Indecision block 802, if all the blocks of content have already beenselected, then the component completes, else the component continues atblock 803. In decision block 803, if the rendering time differential is0, then the component continues at block 806, else the componentcontinues at block 804. The rendering time differential is calculated bythe receive rendering time message component and adjusted by thiscomponent as the rendering of the content is adjusted. In block 804, thecomponent adjusts the selected block to account for the rendering timedifferential. For example, if the content corresponds to videoinformation, then the component may remove frames to effectively speedup the rendering or may duplicate frames to effectively slow down therendering. In block 805, the component adjusts the rendering timedifferential to account for the adjustments to the selected block. Forexample, if the block corresponds to one second of video information andthe adjustment was to duplicate every frame in the block, then therendering time differential is adjusted by subtracting one second. Therendering time continues to reflect the amount of the content that hasbeen effectively rendered. For example, if every frame is duplicated ina one second interval resulting in two seconds of adjusted content, therendering time would only be increased by one second. In block 806, thecomponent outputs the selected block, either adjusted or unadjusted toeffect the rendering of that block of content. The component then loopsto 801 to select the next block of content.

FIG. 9 is a flow diagram illustrating the process of the send renderingtime message component in one embodiment. The component can be executedupon the occurrence of various events, such as when a timer expires. Inblock 901, the component adds the rendering time of this master deviceto the message. In block 902, the component retrieves the device timefor this master device. In block 903, the component adds the device timeto the message. In block 904, the component then broadcasts the messageto the other rendering devices and then completes.

FIG. 10 is a block diagram illustrating the processing of the receiverendering time message component in one embodiment. In block 1001, thecomponent extracts the master device time from the message. In block1002, the component extracts the master rendering time from the message.In block 1003, the component converts the master device time to the timedomain of this device. In block 1004, the component calculates themaster start time by subtracting the master rendering time from theconverted master device time. The master start time is in the timedomain of this device and represents the time at which the master deviceeffectively started rendering its content. In block 1005, the componentcalculates the slave start time of this device by subtracting the slaverendering time from the current slave device time. The slave start timeindicates the time at which this slave device started rendering itscontent. In block 1006, the component calculates the rendering timedifferential by subtracting the slave start time from the master starttime. The component then completes.

From the foregoing, it will be appreciated that specific embodiments ofthe invention have been described herein for purposes of illustration,but that various modifications may be made without deviating from thespirit and scope of the invention. The rendering devices can becomponents of the same physical device. For example, a DVD player mayhave a component that processes the video content and a separatecomponent that processes the audio. The hardware and software of thesecomponents may result in a difference in rendering speed of the content,and thus the rendering can become out of synchronization over time.Accordingly, the invention is not limited except as by the appendedclaims.

We claim:
 1. A method comprising: sending a first message from a firstdevice to a second device, the first message including a value ST1indicating a current device time of the first device when the firstmessage is sent from the first device to the second device: storing atthe second device a value RT1 indicating a current device time of thesecond device when the first message is received by the second device;sending a second message from the second device to the first device, thesecond message including a value ST2 indicating a current device time ofthe second device when the second message is sent from the second deviceto the first device, the second message also including the values ST1and RT1; storing at the first device a value RT2 indicating a currentdevice time of the first device when the second message is received bythe first device; calculating at the first device a first time domaindifferential DIFF 1 between a time domain of the first device and a timedomain of the second device using the values ST1, RT1, ST2 and RT2; andrepeating the foregoing steps to calculate a plurality of values of thefirst time domain differential DIFF1; and calculating at the firstdevice an average of the maximum value of the plurality of values of thefirst time domain differential DIFF 1 and the minimum value of theplurality of values of the first time domain differential DIFF
 1. 2. Amethod comprising: sending a first message from a first device to asecond device, the first message including a value ST1 indicating acurrent device time of the first device when the first message is sentfrom the first device to the second device: storing at the second devicea value RT1 indicating a current device time of the second device whenthe first message is received by the second device; sending a secondmessage from the second device to the first device, the second messageincluding a value ST2 indicating a current device time of the seconddevice when the second message is sent from the second device to thefirst device, the second message also including the values ST1 and RT1;storing at the first device a value RT2 indicating a current device timeof the first device when the second message is received by the firstdevice; calculating at the first device a first time domain differentialDIFF 1 between a time domain of the first device and a time domain ofthe second device using the values ST1, RT1, ST2 and RT2; and repeatingthe foregoing steps to calculate a plurality of values of the secondtime domain differential DIFF2; and calculating at the second device anaverage of the maximum value of the plurality of values of the secondtime domain differential DIFF2 and the minimum value of the plurality ofvalues of the second time domain differential DIFF2.
 3. A non-transitorycomputer readable storage medium having embodied thereon a program, theprogram executable by a processor to perform a method, the methodcomprising: sending a first message from a first device to a seconddevice, the first message including a value ST1 indicating a currentdevice time of the first device when the first message is sent from thefirst device to the second device: storing at the second device a valueRT1 indicating a current device time of the second device when the firstmessage is received by the second device; sending a second message fromthe second device to the first device, the second message including avalue ST2 indicating a current device time of the second device when thesecond message is sent from the second device to the first device, thesecond message also including the values ST1 and RT1; storing at thefirst device a value RT2 indicating a current device time of the firstcalculating at the first device a first time domain differential DIFF 1between a device when the second message is received by the firstdevice; time domain of the first device and a time domain of the seconddevice using the values ST1, RT1, ST2 and RT2; and repeating theforegoing steps to calculate a plurality of values of the first timedomain differential DIFF1; and calculating at the first device anaverage of the maximum value of the plurality of values of the firsttime domain differential DIFF1 and the minimum value of the plurality ofvalues of the first time domain differential DIFF
 1. 4. A non-transitorycomputer readable storage medium having embodied thereon a program, theprogram executable by a processor to perform a method, the methodcomprising: sending a first message from a first device to a seconddevice, the first message including a value ST1 indicating a currentdevice time of the first device when the first message is sent from thefirst device to the second device: storing at the second device a valueRT1 indicating a current device time of the second device when the firstmessage is received by the second device; sending a second message fromthe second device to the first device, the second message including avalue ST2 indicating a current device time of the second device when thesecond message is sent from the second device to the first device, thesecond message also including the values ST1 and RT1: storing at thefirst device a value RT2 indicating a current device time of the firstdevice when the second message is received by the first device;calculating at the first device a first time domain differential DIFF 1between a time domain of the first device and a time domain of thesecond device using the values ST1, RT1, ST2 and RT2; (a) sending athird message from the first device to the second device, the thirdmessage including a value ST3 indicating a current device time of thefirst device when the third message is sent from the first device to thesecond device, the third message further including the value RT2; (b)storing at the second device a value RT3 indicating a current devicetime of the second device when the third message is received by thesecond device; and (c) calculating at the second device a second timedomain differential DIFF2 between the time domain of the first deviceand the time domain of the second device using the values ST2, RT2, ST3and RT3; repeating the steps (a), (b), and (c) above to calculate aplurality of values of the second time domain differential DIFF2; andcalculating at the second device an average of the maximum value of theplurality of values of the second time domain differential DIFF2 and theminimum value of the plurality of values of the second time domaindifferential DIFF2.